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The ais of' this study was to show that the 
•nperlority of correctea-for-<|nessing scores over naiber right scores 
as true score estimates depejids on the ability of examinees to 
recoani^e situations where they ca?* elimiivate one ot more 
alternatives as. Incorrect and" to omit items where they would only be . 
guessincr randomly. Previous" investioations show examinees lack this 
ability, ftn instructional strategy for teaching examinees to use 
partial information in takiner multiple choide tests under formula 
scoring^ conditions was tested on 2 BO fourth, fifth and sixth grade, 
private end public school students from' the Onited States Virgin 
Islands, "^hl* method Is outlined and followed by sections dealing 
jrith the administration of pre-tests and post*tests* and the provisidn 
of training to selected groups (in the form of an algorithm for 
attftcking'-test Items, and practice sessions with immediate feedback! • 
It i^^.dicated id the final discussion of . research results: that the 
technigue wa% successful in training examinees to omit only items for 
which thev had no more than a chance probability of correctly 
respondinq, Peferences, figures of the algorithm and pr.e-test 
practice items, and tables of descriptive statistics are appended, 
(Author /JIEP> 
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_ Abstract 



The superiority of corrected-for-guessing scores over* number right scores 

I. ^ 

as true score estimates depends on the ability of ^aminges to recognize 
^^ituations where they can eliminate one or more alternatives as Incorrect 

and to omit it^ms where they would only be guessing randomly. Previous ' 

investigatipns show examinees lack this ability. /.This study tested an 

instructional strategy for teaching examinees to use partial information ; 

in taking multiple choice tests undeV formula scoring conditions. » The^^^v 
* results indicated' that the technique was sticcessful in training examinees . 

to omit only items for which they had no more than a chance probSblTity 

of correctly responding. 
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• ; ' ". • J- ' - ■ ' ' /' , ' * 

The purpose of this study was ;. to test the effectiveness of a' training v 
■'• ■ ' ' ■ ^ , 

strategy designed to teach' students, to use partiaT. information which Ithey 

ma^^^have at .their disposal when answering:! tenis on multiple choice te§ts,^ 

Lord (1975) showed that using^ the correction for guesfing in adminis- ^ 

tering and scorijig mul tipTe choice tests would provide scores which would" 

have higher levels of rel iability and validity than scores obtained using 

number right directions.* Number right procedures, Result in a score which 

is the actual numb'er of itirhs th? examinee marks' correctly, while the L 

, . . ■■ ^ - • • ^ \ 

formul^j score- is this number of colrrect it^s minus a fraction, of the num- 

ber marked incorrectly (usually w/C^ where W is the- number loarked wrong 

and C is the number of choices per item)v In the former, situation' examinees 

are directed to answer all items while in the latter tbey^are directed. to u 

leave an item blank if they can make no better than' a random guess , at the . ^* 

answer. The superiority .of the formula scoring method is dependent' on the 

assumption that the examinees follow these directions and OTiit only items 

for which they w6uTd have to guess from among all choices. . Examinees' 

who could eliminate one or more choices as incorrect (i.e. had some partial 

• * "- 

information concerning , the item) should guess from among theij;;emaining choices. 
However^ numerous studies including^those by Sheriff^ and Boomer (1954); 

Crehan» Candor, & Beckett (1976), Slakter, Juliano, & Sarnacki \ 1976), Cross 

, . , . * . ^ 

and Frary (1977), Pbgg.lo, Amus', &• Levy |l978), and Bliss (1980) have shown 
that eJ&minees omit iti^ms that they had a better tUn chance probability, 
of ar^wering cprrectly and were, therefore, penalized by the use of the 
correction for guessing formula. This phenomenon has been observed acrorss 
grade levels from elementary school through university levels. -Further, 
81 Iss^ noted that, in the tntermediate grades (4 through' 6) it was the 

_ / ' ■ .-4, • 
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higher achieving students who tended to report having followed formula 
scoring directions and were, therefore, penalized most often. 

It is possiblje that formula scoring directions (answfer if^ou know any- 
thing omit if you know nothing ) make unreasonable demands on'the^self- 
knowledge af test takers. If this is so, formula scoring techniques would 
tend to be inappropriate for any multiple choice test. In th':s case, testers 
would be forced to fall back^on number right scoring techmqufis with Us 
accompanying decrease in score validity or rely on other, 'rel at fvel.y w^tested, 
scoring techniques (see Frary, 1980). However, Bliss noted that tnefe were 
some students who did report complying with formula scoring directions 
and who seemed to be able to make these distinctions. This suggests that 
the aforementioned ability may be a learned skill and that it may le 
possible to' teach this skills to examinees prior to their taking tests using 
formula scoring directions. Ihe purpose of this study was to test the 

effectiveness of such a;^ teachi-ng. strategy. 

• • • * ■ 

' . . Method 
Subjects ' ■ ^ t 

A sample of 140 students, Wfi's randomly selected from the fourth, f"5fth, 
and sixth grades of two publ ic schools in St. Thomas, U.S. Virgin Islands 
and an additional 140 from two St.' Thomas private schools for a total of 
280 subjects. Members of each of these groups of 140 were randomly assigned' 
to one of the four groups^ of a Solomon Four Group desigq (see CampbelT arid ^ 
JtatileS^, 1963). yielding 70 subjects in each group. 
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Pretest directions and administration 

A test composed of 36 randomly selected items from the Mathematics*- . 
Concepts subtest M-1 of Form 6 of the Iowa Test of Basic Skills ^Kieronymaus 
^nd Linquist, 1971) was administered , to two of the four student groups '-""^^ f 
{designated "pretest groups"). This form' is designated as being most 
appropriafe for sixth graders and had never been used 'in any of the schopT§ , 
in which the sample students studied. Testing two thirds of the subjects 
above grade level was. done in order to obtain larger numb,ers of omitted 
' responses than might have been ohitained were subjects tested at grade level. 
The test directions were projected on a large screen and r^fad-ailoud at 
the same time. Jhe portion of those which involved the method of /b^^oring 
-to be used and' the manner of scoring are presented below. 

Your score will be the number of questions you answer 
correctly minus/fhe number you answer incorrectly* . 
You should answer questions even when you are not sure 
' that your answer 'is correct. This is especially true 
if you know one or more of the- choices is wrong or if 
you^havfe a "feeling" about which answer Is correct. ^ 
However, it is better to leave a blank than to guess 
• ' wildly. . I - ' ; " : ' 

These directions are taker, from Davis (1967) and were found to be under- 
standable to fourth, fifth, and sixth grade students by Bliss (11980) 

Examinees were directed to turn in their answer sheets and^^thelr test 
papers when they were finished. They were then given a reading assignment, 
unrelated to the test content, to do while ^other examinees completed" the 

. • ••■ ■ ■ - ^ . ^ 
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test. At no time were they allowed to look up answers to the test or allowed 

to discuss the test. with /their fellow examinees. 

After all ex^^minees had completed -the pretest (never more than 35/ 

minutes) -they were asked to put away .the reading assignment and their answer 

' sheets atnd test apapers were returned. They were thf.ti given red"*pe'n5 and 

told th^t their ^scores would actually qdnsist of .the total number of 'rtems 

t^ey marked correctly, regardless of color, and were' directed to answer 

all the it^ms they left btsn!* ^ven. if this meant taking a wil.d guess. Examin- 

ees were per .fitted to wonk until, they had comple-ted the task. 

Training procedu re's 

^ One of the pretested groups artd^one of the •ribn])retested groups was 

chosen, on a chance basis, to receive the training strategy^ This consisted 

of three hours of instruction conducted in three one hour sessions, each 

one on5 week apart. The first session occurred lone week after pretesting 

was completed. The training strategy consisted of three phases, each taking 

* . • . "J 

approximately one hour. ' 

Phasa I consisted of expl^iining the process of fbrmul a scoring^ the 

presentation of an algorithm for^ttacKing muHiple choice questions, and 

the presentation- of the algorithm -,in the form of, a flow chart (Figure IJ., ^ 

Examinees were then provided with sample multiple "choice Items, in increasing 

' difficulty, to use in practicing the use of the algorii^m. Their levels 

of success were recorded so that.eadh student could cocnpare earlier and 

later. efforts (see. Figure 2J . • * 

Phase II involved the use of latent image^printing techniques. Students 

were provided with collections of multiple'choice jtems, arranged in order of 
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increasing 'difficulty, with boxes to the left of eacTh alternative, Printed 

" ■ ■ ^ ■ * * 

in each box was a "C"*if the answer was correct or.a "W" if the ans)ver was 
wrong using a latent image process in which the letter v/as invisible until 
the .box was colored in with a special marker. Students were directed to 

/Use the item attack algorithm to fill in the boxes' of ajl items they 

, ■ > . \ . 

btilieved were incorrect until they marked the* box labeled "C" or until they 

were left with a single choice. The procedures were essentially those 

described by Pressy (1950). In this manner, students received immediate 

feedback*as to the\r success in using the algorithm. 

Phase III involvyed" the taking of two short practice teists under 

formula scoring conditions. 'Tests were sglf-scored by students who were 

then required to fill an" items they left blank.. These were also self-scored 

and the proportions of blank items marked correG:€ly were listed on a black- 

.„board^and compared to, the expected values. This procedure of testing and 

self-scoring was repeated again, the results listed, and the results of 

tilis repeated testing were compared with the results of jthe first. 

Posttesting directions and administration ' ' ^ , 

/All fcur groups were administered a^ 36 item test composed of questions 
/ ^ . . . 

from Form 5 of the Mathematics Concepts^^subtest of the Iowa Test of Basic 

Skills . These were analogous tjD the Items chosen* Vrom Form 6 and used on 

the pretest. Identical directions and procedures were used in the pottesting 

■ " . ii . \ . • ■ ■ 

situation as were used in pretesting. Post'tests. were administered three 
•» ■ ,. . ^ ■ . ^ . 

weeks after the last training session. j 
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Results 



Descriptive djtta^ - * ,^ . 

. Descriptive' statistics for the test scores of the" 242 examinees who 
omitted one or. more items "are presented in Table 1.- The test taking 

behavior variable was examined In two ways. First, the correction for 
guessing formula was applied to the items omitted under the first set of 
direfetions*. If examinees were using partial information .as .directed, the 
expected value of these "red-formula scores" would be zero.. Second,^ the" 

w , n • » 

proportion of those items prviously omitted^" but then guessed correctly was 
determ^ined. The expected value of tFiese scores would be .25 since there . 
were four choices for each item on the test. The^e two measures would 

only be linear transformations of each other if each examinee had omitted 

.* , »>. 

the same number of items. Table 2 presents the red-formula scores and 

.• ^ • . . - 

proportion correct for each achievement test*a.dmin1stration. , , 

Effect's of pretesting * - . ■ ^ 

' • * ' - * . . . 

A 2x2 analysis of variance using unweighted means techniques .'carried 
put using pretesting (group pretested or not pretested) and the presence , 
or absence of the training procedure as the independent variables revealed 
*no sijgnificant main effects for, pretesting or pretesting by treatment inter- 
actions at the «=.05 Ipvel Ydr posttest proportion guessed correctly. 
However, there were. slight (\01 < « < .05) main effects for pret;esting, bu^t 
no pretesting by treatment ijjteractions using red-formula posttest scores as 
-the dependent variable (Tables '3 and 4). While this s ign if fcant pretesting 
effect was disturbing, the lack^of any interactions and the absence of , 
pretesting main effect on the other critereon measure produced the decision 
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tb continue the analysis pooling posttest scores, on both variables. , 

• ■ ^ 

Eff ects of .training ' . . ^\ 

\ • ^ ' ' 

Table 5 contains pretest and posttest red-formula and proportion guessed 

correctly scores. In neither of the pretested groups does the 9b% confidence 

interval contain thie expected values of eiiher of the critereon measures." 

Hence;. it'does not appear that either of the pretested groups were using ^ 

partial information in J:he manner^perscribed by the test di recti ons'pr^^^ to 

training. Posttest mean scores for both variables, however, show the expected 

values within the 95% confidence intervals fojr the groups receiving the 

.training on using partial information. Ttle mean posttest values^ for red- 

formula and proportion guess correctly scores arei .significantly higher than 

the expected values (0 and .25, respectively) .in the untrained group as 

indicated' by the fact that the 95% corifidence intervals for these means 

do not include the expected values. " " ' ' * 

Thes^data indicate that during pretesting e){aminees, on the average^ 

tended to omit items under formula scoring directions which they had a 

better than chance probability of ^tnswering correctly. On the posttest, 

however, the trained group appeared, on. the; average,, to^ omit only those 

items for which they could respond with no better than a random guess. The 

' " " - / . - ;'■ ■ , .. 

untrained gr:oup of examinees continuedHo omit items for which they could 

have responded^ with more than simply a random guess. ^. 

Discusr . , ' * * 

The training .procedure, designed on the principle ^f t)resentation 

of an^algorithm for attacking; items with subsequent practice and immediate 
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feedback, appeared to be an effective strategy for training^^examirfees to 

1 . & . 

use partiatl information when taking multiple choice tests. This is 

' # ■ . • 

encouraging for a number of reasons. Lord (1975) has shown, theoretically, 

that when examinees use partial information in \he mann&T*- described herein, 

formula scores are more valid indicators of true score than are number right 

"scores. Indeed, users^of testing results assume i^hat examinees are, making 

their best' efforts when taking tests, '^This best effort must include^making 

use of all information available when responding to test questions. By 

definition, then, examinees who are not using partial information are' 

not making their best.efforts and this should have a negative effect op 

t * : ■ ' ' ' f^. . 

I ' 

the validity of obtained scores. , ' . 

Second, the results of this study 'seenj ito indicate that the ability 

to use partial information in taking mul.jtiple choice tests seems to Be a 

■ \^ ■ ^ • 

skill that can be learned and learned rather quickly and effectively^? If the 

■ ^ ^ * . / . • 

results of this study were to be validated over various populations, schools 
could include such instruction as part of their basic curricula. 

This study generates a number of unresolved questions', however. First, 
'there is the question of- the stability of this training oyer time. Since"^^ 
the time between the last training session and the posttest was only tfjree 
weeks, there is non'nHTcation that examinees will retain this skill over 

time. Second, while tests ort^ the means of group scores showed gain's in.. the 

■ ■ ' ■ ■ . • • ' . ■ 

predicted direction, the variances of scores on both criteredn meas^ui:ies__wasv-- 

certainly not zero. There are differences in the efffects of training' 

» * ' . ", ■ 

among examinees. Further work Ineeds to be done to examine^ examinee • 

- * . ." " •', 

characteristics that may interact with the training procedures. The authpr 
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^ . V. ' 

suggests that a good place to begin looking for these could be^niong those 
variables which tend to interact with achievement, in- general i including. ; 
such things as past achievement (see Bliss, 1980), conceptual level . ^ Hunt, 1970) 
IQ scores, and socioeconomic status. ; 

Finally, the results oif this study need to be replicated among other 
populations where previous studies have shown an inability of examinees tb 
use partial, information. This study can be viewed as the first of a 
'series to investigate the effectiveness of a particular teaching strategy. . ' 
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Achievement Test Pre- and Pos;t'test Statisti 



cs 



Group 



Pretest 



Posttest 



.Fbrmula score Number right 



Formula -score Number right 



- KR--^0 , X SD KR-20 • X SD 



KR-20 X SD KR-20 



Pretest/Treatment 
_ (n=60) - 



16.83 8.10 . .88 20.88 8.66 .90 



20.88 7.10 .92' 24.93 8.87 .89 



Pret6$t/Control 
. (n=68J 



•17.63 8,82 .90 ii2,2p 7.03' .84 



18.87 7.53 .90 ^76 6.2>.. 



88 



Nonpretest/Treatment 
ln=55) 



20.51 7^77 ,91 24.85 9.62 . 



89 



^ TABLE 2 

Summary of Red-Formulji and Proportion Correct Scores 



Group • 



• Pretest/Tre^itJnent 
(n»60) 

Pretest/Qontroi 
(n-68) 

Nonpre tes t/Tre atmen t 
(h-55) / 

Noiiipre test/Control 
/ (n»59) 



Pretest 



Red-Formula Prop. Cor. 



X SD 



SD 



Posttest 



Red-Formula Prop. Cor. 



X SD X SD 

» f 



.74. 'l.e! -,35 .29 .17 i.ii .25 



1.09 1.78 -.37 .27 



.21 



1.21 1.81 .39 .27 V 

.05" .92 .28 .15 

\ . • 

.56 1.56 >33 .30 . 
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Table 3 
Analysis of Variance Source 



Table 


for, R'ild'-Fori^ila 


Scores 


- 


Source 


ss • 


df 


MS 


F 

- ^. 


*• * 
Pretesting 


9.738 


1 ' 


9.758,. 


4.93* 


Training 


. 34.972 


1 


34.972 


17.67* X 


Pretesting X 'T*raining 




1 — 






Within cells 


471.039 


238 


1.979 . 




p<.05 








« 

t 


■ .1 








■ ' *> 




Partial Information 
16 



Table 4 

r 




Analysis of Variance So;^rce 
Tajb^e for Proportion Guessed Correctly 



Bovirce 


' SS 


df 


-Ms' 




Pretesting 


.061 


1 


\oVi 


1.1 5> 


/Training 


.303 


1 


. 303 


5'.71 


Pretesting X Trai^itag 


.182 


1 


.182 


3.43 


vnrtHm ceTls - 


12.518 


. 238 


.053 





* p<.05 
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. . A TalDle'5 , 

. Pooled Pi;-etest an<^ !Pcfsttest Red-Formula 
and {proportion Cprrect Scores 



N 



95% confidence 
interval 



Pretest 

Red-Formula Scores 

Tr^^tment Group 60 
Nontreatment Group 68 
. Proportion Guessed CozrectJLy 

Treatment Group 60 
Nontreatment Group 68 



.742 
1.090 

o .351 
.365^ 



,326 tp 1.163 
,663 to 1.512 

,277 to .425 
301 to .429 



Posttest 

Red-Formula Scores 

Treatment* Group 115 

Nontreatment Group 127 
■ Proportion Guessed Correctly 

Tr e a tinen t Group 115 

Nontreatment -Group ; 127 



U4 - 



'910 



.275 
.361 



.076 to .904 

.618 to 1.202 

.241 to .309 

.307 tp .414 
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Figure 1 

Flow Chart of the Item Attack Algorithm 




Read Question 



N ■ / 



Leave the 




ITEM BLANK 


No . 










Yes 




Eliminate these ' 

CHOICES • 






Guess Vrom among the 
other choices 
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Figure 2 
Model Phase I Practice Page 

The capital city of iowa is 

A. ALBAf^Y , ' ' 

B. Des Moines " . ' 

c. Souix City- ' , 

D, Washington/ D.C. , 

1. Do YOU KNOW THE ANSWER? 

Ai Yes - Write the letter of the answer here. ' 

B, No - 60 TO STEP #2 - - 

2. Can YOU eliminate any of the choices as being wrong? 

. Av Yes List the letters of the wrong answers here. 

:>( ' ■ . , ■ • ' ^ 

: Now^ take a GUESS* from among the choices you don't know 

ARE WRONG AND LIST THKANSWER HERE. 

.<>' ■ . • . % * * 

B. No - 60 ON TO the NEXT quest I ON J 



